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ABSTRACT 



Cluster evaluations have been defined as involving multiple 
sites, focusing on longer term projects, involving substantially different 
approaches to similar problems, and improving the social condition. Cluster 
evaluation is illustrated through the description of an initiative funded by 
the W. K. Kellogg Foundation called the Leadership for Institutional Change 
(LINC) program. LINC seeks to provide support and guidance for new leadership 
models in higher education among land grant universities. Cluster evaluators 
were identified rather early in the LINC process, but only after the 
extensive needs analysis had been conducted. The cluster evaluator did have 
some input into the process of initiative design. Experiences from the LINC 
initiative and other cluster evaluations have resulted in a proposed cluster 
evaluation framework, which contains the following elements: (JL) evaluate the 

planning context and client system; (2) assess the needs assessments (3) 
evaluate and refine the program objectives; and (4) assess the proposed^ 
objectives and action plans. (SLD) 
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Introduction 



What is the role of cluster evaluation in the life-cycle of a program or a project? The 
answer is, it “depends” This question looms larger than normal when a project is billed as a 
“learning” opportunity or an organizational change project. This paper is a way to help me get a 
bearing on just where I am in the evaluation/program development landscape. Some may likely 
think that cluster evaluation is over the horizon or at least not in the neighborhood of good, solid, 
trusted, and time tested traditional evaluation projects. Perhaps they are correct, however, when 
viewed as evaluation of developmental undertakings, cluster evaluation may be messy, but it is 
vital to the growth and development of initiatives. 

Cluster evaluation has been defined elsewhere (Schmitz, 1997; Worthen and Schmitz, 
1997), but a brief review may help us gain some common ground. Cluster evaluations have been 
defined by the following characteristics: 1. Involving multiple sites, 2. Focusing on longer term 
projects, 3. Involving substantially different approaches to similar if not the same problems, and 4. 
Improving, on a rather large scale, the social condition. 



ERIC 



2 



Cluster Evaluation as Program Development - Byron R. Burnham 



Background 

Given this rather wide definition of cluster evaluation let me set the stage for a particular 
cluster evaluation. The W.K. Kellogg Foundation a number of years ago funded an initiative 
titled Food Systems Profession Education which was aimed at helping colleges of agriculture in 
the U.S. to think about their futures and to undertake necessary changes needed. As that initiative 
developed a number of things emerged. Among them was the need for a new look at leadership 
among the nation’s land-grant universities. Hence the birth of another initiative “Leadership for 
Institutional Change” (LINC). As described on its web page, LINC is... 

A national leadership development dialogue established by the W.K. Kellogg 
Foundation among twelve state and land-grant university consortiums throughout 
the United States. Each of these twelve consortiums are also involved in the W.K. 

Kellogg Food Systems Professions Education (FSPE) Initiative. The FSPE 
initiative seeks to catalyze change in state and land-grant universities throughout 
the united States. In order for that change to be successful, new models for 
leadership within land-grant universities are required. This Leadership for 
Institutional Change (LINC) Initiative seeks to provide guidance and support for 
new leadership models in higher education. 

(http://www.kellogglinc.com/about.html) 

While the description does little for someone looking for evaluation guidance, its stated 



outcomes offer some direction. These are listed as: 
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1 . Enhance the capacity of the W.K. Kellogg Foundation to support systemic change. 

2. Support the development of values-based, enabling leadership that is both a cause and a 
result of transformational change. 

3. Facilitate change in higher education that better connects colleges and universities to 
the needs of learners and communities. 

A little more help can be found in additional hoped for outcomes: 

1 . Collective leadership models that focus on faculty and their external partners working 
in collaboration for the good of the institution and stakeholder groups. 

2. Development of a critical mass of faculty and partners with shared leadership skills 
built on shared visions and values. 

3. Catalyzing successful long-term sustainable models that are responsive to those 
institutional changes advocated by the resident FSPE project. 

4. Sustainable models of leadership development that allow universities to continue to 
build a supply of sophisticated, well-trained leaders for the future. 

With these stated outcomes, most evaluators would be happy as clams and move ahead 
with plans to measure the above listed outcomes. But even with these outcomes there can be a 
fog and questions about what a funder “really” wants. These kinds of questions, which stem 
from the nature of the initiative and which have to do with organizational change and all that it 
implies, include: changing organizational culture, building a new set of skills among faculty and 
administrators, and changing traditions. When one begins a project with outcomes like enhance, 
development, facilitate, and catalyzing, I assure you that things can and do get cloudy in a hurry. 
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Whose fault is such a situation? How does one deal with fog? What steps can be taken to 
make more certain that such problems do not occur in the future? I would submit that these are 
the wrong questions. 

The questions we ought to be asking, especially in a cluster evaluation that deals with 
changing organizations, should deal with learning and sharing. “What did we learn from this 
project and project activities? and “how can we best share that with the appropriate audience?” 

To help answer those kinds of questions it might be best to look at cluster evaluation that 
deals with organizational or institutional change in the context of program planning or program 
development. Such a look will provide some “hooks” or points of contact for cluster evaluators. 

Program Development 

A quick review of the program development literature will help further contextualize 
cluster evaluation in program development terms. Most of the references in this section come 
from discipline of adult education. Program development is contrasted with instructional design. 
As a result, the concepts discussed are on a much wider breadth than instructional development 
terms would provide. For example, needs analysis in program development seldom refers to skill 
development, rather it reflects audiences analysis, program content, delivery formats and the like. 

Sork and Caffarella (1989) offer an interesting and relevant comment on program 
planning. “In its most general sense, planning refers to the process of determining the ends to be 
pursued and the means employed to achieve them” (p. 233). This is descriptive of the initiative 
that I’m currently involved in as a cluster evaluator. The parts of the planning model described 



Cluster Evaluation as Program Development - Byron R. Burnham 



by Sork and Caffarella (1989) include: (1) analyze planning context and client system, (2) assess 
needs, (3) develop program objectives, (4) formulate instructional plan, (5) formulate 
administrative plan, and (6) design a program evaluation plan (Sork and Caffarella). Other 
functional concerns include participation, staffing, resources, coordination, and external relations 
(Knox, 1991). Using this framework of program development, I would like to briefly describe 
cluster evaluation functions with each category of program planning. 

Analyzing the planning context and client systems 

Sork and Caffarella have outlined some important considerations that could very well help 
cluster evaluators and project or initiative personnel. Areas of concern listed include: (1) the 
history and traditions of the organization, (2) the current structures that govern the flow of 
communication and authority, (3) the mission of the organization, (4) the resource limits, (5) the 
standard operating procedures, and (6) any philosophical constraints that limit who can be served 
or what types of needs can be addressed. These factors are internal to the particular organization 
that is doing either the project or initiative planning. They are important considerations as 
initiatives are being thought about. These elements might well be used as preliminary screens in 
selecting potential grantees. They might make more sense for selection purposes than would a 
tightly planned and proposed project that lists outcomes which may or may not be appropriate or 
even possible in the very preliminary stages of planning. At very least they are important areas for 
study by cluster evaluators especially when initiatives aim at the heart of institutional ways of 
doing business. These elements can help at two levels: as a pre-screen for funders; and as a guide 
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for cluster evaluators as projects proceed. 

Needs analysis 

At an initiative level the Foundation does an exceptional job of needs analysis. Often 
months and even years of fact finding, discussions with various clients and stakeholders, and 
formal and informal program offerings inform the Foundation of problematic areas which leads it 
into initiative areas. Cluster evaluation may be considered at this point, but often the cluster 
evaluators are not in place until after the needs analysis has taken place. In the particular case of 
LINC, the cluster evaluators were identified rather early but after the extensive needs analysis. 

The cluster evaluator had some initial input into the process of the initiative design. 

There is another level of needs analysis that is perhaps less focused upon at the initiative 
level. The individual project level is more specific in terms of outcomes and objectives. When a 
funder offers dollars for participation in an initiative, potential projects propose activities and 
outcomes that are intended to fit within initiative’s broad goals. Using the characteristics listed in 
the “Contextualizing” section above, the potential of mismatch between projects’ and initiative 
goals may be reduced. After the fact (after a project is funded within an initiative area), this is one 
area where a cluster evaluator might look retrospectively and ask about the initiative’s need 
definition and the project’s need definition to determine the relationship. This activity could help 
reduce the comments that are often heard that go something like “what does the Foundation really 
want?” or “what is the real agenda here?” Such comments have been encountered in the last two 
cluster evaluations with which I have been involved. This is likely a function of just how 
developmental the initiatives really are. 
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Program objective setting 

This is the life line of evaluators. There has been much debate about the value of an 
objective vis a vis evaluation and it usually comes up as being very important. I need objectives, 
goals, outcomes to do my work as an evaluator. Sork and Caffarella define program objectives as 
“statements of the anticipated results of the program” (p.238). They divide objectives into two 
broad categories: educational objectives which deal with participant learning and organizational or 
operational objectives related to educational for organizational function. Objectives are rooted in 
the needs analysis and provide benchmarks or comparisons for cluster evaluations. 

But there is debate about objectives in terms of program planning. Reliance upon set 
objective often limits what the cluster evaluator can and should look for. Unanticipated outcomes 
or objectives that evolve or change or that are added can and will be missed. In fact, in a 
development initiative perhaps the critical function of objects are to merely “point the way” or 
give direction and not provide the foundation of careful measurement. 

Terms like enhance, support and facilitate are rather difficult for an evaluator to get his or 
her hands around. Measurement becomes problematic and hence providing evidence may be 
difficult. I would submit that the kinds of goals mentioned earlier are an indication of a 
developmental approach to a problem. It suggests a lack of definition at the beginning of an 
initiative. The goals may actually tell us more about the program if we ask an important question: 
“Why are the goals so global at this point?” Global goals can provide expectations for cluster 
evaluators. The contribution from a cluster evaluator might not be tight measurement, but rather 
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refinement for an initiative and its projects. Such refinement comes as the cluster evaluator, 
funder, grantee, and others interact to better define initiative outcomes. 

Learning activities 

The program development model presented by Knox (1991) and Sork and Caffarella 
(1989) is fixed in a learning paradigm and thus does not necessarily view program development as 
having other outcomes beyond learning. In today’s setting much has been written about learning 
organizations. For example, in Sculpting the Learning Organization by Karen E. Watkins and 
Victoria J. Marsick, the authors provide a whole new arena for program development activities 
that are more organizationally bounded and influenced than a strict learning experiences would be. 
And it is into this arena that the Foundation has lead this cluster evaluator. 

If learning is considered in an organizational frame, then additional dimensions of 
measurement open to an evaluator. Organizational capacity becomes a more refined concept and 
new ways to document change and progress become possible. For example, organizational 
openness, communication, cross functional teaming and more can all be closely looked at. 

Looking for additional documentation approaches is in keeping with the sage words of the 
sage words of Connie Schmitz (1997) which become not only descriptive, but also salutary. 

Not every program cluster (or its evaluation) is expected to conform... in 
terms of scope, structure, or strategy. Each time the Kellogg Foundation launches 
an initiative, it must wrestle with a number of issues that are unique to the nature 
of the problem begin addressed, the participants involved, and the initiative’s 
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specific mission and goals. Perhaps the single most important principal embedded 
in it is that a successful cluster evaluation depends upon a team 
approach.... (Schmitz, 1997). 

What is described here is an attempt by cluster evaluators to keep up with a changing 
evaluation context. It is not uncommon for a cluster evaluator to ask program staff “what new 
questions do you have based on what you have heard and what information we have assembled at 
this point in time?” To hold to concrete, specific, and unchanging objectives would not only 
provide old, stale and potentially useless information, it would also fail to describe what the reality 
of a changing initiative might be. If we subscribe to the notion of useful evaluations then, in a 
cluster evaluation situation we may have to worry less about contextual terms and more about 
useful information. 

Program Evaluation 

This arena is duck soup for an evaluator, or is it? Evaluators can move quickly to several 
models and choose, or they can pull from existing models to build one expressly for the evaluation 
situation. But this is just precisely where the rub comes. The selection of the model, or parts 
thereof can lead to evaluation paralysis. We can become so worried about doing it properly, that 
we easily forget to do anything. Cluster evaluation, at least in the context which I have described 
(which is one of evolving and developing initiatives), has to be fast, timely (which is different from 
fast), accurate, instant, and helpful. 

Connie Schmitz and Blaine Worthen have suggested that some degree of structure be 
superimposed upon the sometimes chaotic environment of cluster evaluations. On the surface, this 
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recommendation has much merit, however, when looks more closely at the cluster evaluation 
environment, this may not be possible. In fact if the cluster evaluator is doing his or her job, then 
he or she may actually add to the apparent chaos. Patton (1978) used a very descriptive term as 
he wrote about evaluation and utilization. That term, the “active-reactive adaptive evaluator,” 
(p.284) in a nut shell describes the role of the cluster evaluator in the early developmental stages 
of an initiative. It is extremely important for a cluster evaluator not to get hardening of the 
methodologies. Just as critical, the cluster evaluator needs to understand that his or her role will 
change over the course of the initiative. Early on it is being a scout, an early warning system, a 
canary in a cage in a mine. But the role may change into one of measurement, of proving, and of 
testing. Flexibility is an important characteristic for cluster evaluators. 

A Proposed Cluster Evaluation Frame 

Cluster evaluation as program planning may well provide the structure that Schmitz and Worthen 
call for and that I earlier noted may be difficult to obtain. Cluster evaluation may have to go 
through the outlined program development steps if it is to be useful to the people involved. These 
program development areas are next listed with potential questions that a cluster evaluator may 
want to ask. 

Evaluate the planning context and client system 

What evidence is there that the history and traditions of an organization match the overall goals of 
an initiative? Who are the decision makers and influencers in the organization? How well does the 
mission (or past mission) relate to the initiative? Who is the organization’s audiences? 
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Assess the needs assessment 

To what degree has the needs analysis been conducted? Who did the needs analysis? What were 
the sources of information? How broadly have the results been accepted? 

Evaluate and refine the program objectives 

How firm are the project’s objectives? Who was involved in establishing the objectives? How 
widely accepted are the objectives? How likely is it that the objectives will change? And how 
quickly will they change? Which kinds of objectives are prevalent (educational objectives or 
organizational/operational objectives)? What organic benchmarks are available for use? Do the 
objectives need further refinement? Who will do this? 

Assess the proposed objectives and action plan(s) 

Who owns the vision? What is the organizational purpose? How well does this initiative fit the 
organizational purpose? Is there resistance to the project/initiative? Where is it coming from? To 
what extent did team members define and develop objectives and action plans. 

Comment upon the proposed project evaluation plans 

How fixed or firm are the evaluation activities? Are there multiple evaluation models being 
proposed for the evaluation plan? 

Answers to these questions will not only help steer the cluster evaluator, but will also 
supply important information for cluster evaluation reports. This framework or one similar to it 
may well help cluster evaluators navigate unfamiliar territory or territory that is changed by 
upheaval in the institutional tectonic plates. 



ERIC 



12 



Cluster Evaluation as Program Development - Byron R. Burnham 



References 

Knox, Alan, (1991). Educational leadership cmd program administration in Adult 
Education: Evolution and achievements in a developing field of study. Peters, John M. and 
Jarvis, Peter and associates. Jossey Bass Publishers. San Francisco. 

Patton, Michael Q. (1978). Utilization-focused evaluation. Sage Publications. Beverly 
Hills, CA. 

Smitchz, Connie (November, 1997) What would an ideal cluster evaluation look like? 
Paper presented at the 12 th Annual Meeting of the American Evaluation Association, San Diego, 
CA 



Worthen, Blaine, and Smitchz, Connie (1997). Conceptual challenges confronting cluster 
evaluation. Evaluation, 3(3):301-320. London: Sage Publications. 

Sork, Thomas J. and Rosemary S. Caffarella (1989). Planning programs for adults. In 
Handbook of Adult and Continuing Education, (Sharan B. Merriam and Phyllis M. Cunningham 
eds. Jossey Bass Inc., Publishers. San Francisco. 

Watkins Karen E. and Marsick Victoria J. (1993 ). Sculpting the Learning Organization. 
San Francisco, Calif. : Jossey-Bass 



12 




13 



TM030307 



U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

~ Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 




II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, If 
reproduction release is granted, one of the following notices is affixed to the document. 





If permission is granted to reproduce and disseminate the identified document please CHECK ONE of the following three options and sign at the bottom 
of the page. 




Level 1 Level 2A Level 2B 

1 t t 






Check here for Level 1 release, permitting reproduction Check here for Lewi 2A release, permitting reproduction Check here for Le wi 28 releas e. 

vtd dissemination in microfiche or other ERIC archival and dissemination In microfiche and in electronic media reproduction and dissemination In microfiche only 

madia (e,g., electronic) and paper copy. for ERIC archival coflection subscribers only 

Documents will be processed as Indicated provided reproduction Quality permits. 

If permission to reproduce is grwted. but r» box ts checked, documents wUJ be processed at Level 1. 



Sign 
0 h *re,-* 

ERJC ease 



/ hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this document 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system 
contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agencies 
to satisfy information needs of educators in response to discrete inquiries. 




im Old Main 



i wmm fteowtes/ 

tim .if. gM 








III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, If you wish ERIC to cite the availability of the document from another source, please 

— provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly 

— available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 

University of Maryland 

ERIC Clearinghouse on Assessment and Evaluation 
1129 Shriver Laboratory 
College Park, MD 20742 
Attn: Acquisitions 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 

ERIC Processing and Reference Facility 
1100 West Street, 2 nd Floor 
Laurel, Maryland 20707-3598 

Telephone: 301-497-4080 
Toll Free: 800-799-3742 
FAX: 301-953-0263 
e-mail: ertcfac@ineted.gov 
WWW: http://ertcfac.plccard.csc.com 
O 

E RIO-088 (Rev. 9/97) 

auuflam EVIOUS VERSIONS OF THIS FORM ARE OBSOLETE. 



